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ABSTRACT 

Reconsiderative scoring occurs when student responses are received and reviewed by the 
teacher before points for correctness are assigned. Manually scored completion-style 
questions are reconsiderative. A new method of machine assistance produces on a 
microcomputer an item analysis that prints the actual word response or numeric answer. The 
teacher reviews the respon!;es prior to allocating points via the keyboard. Initial experiences 
are reported. Pedagogical implications, including additional software capabilities to assess 
higher order learning, are presented. Reconsiderative scoring could open a new dimension 
in educational measurement for both teacher-generated and standardized assessments. 

[Paper presented at the joint conference of the National Council for Measurement in 
Education (NCME) and the American Educational Research Association (AERA), Boston, 
MA, 16-19 AprU 1990.] 

• • * • (A one-page synopsis is on page 26.) 



I. Concepts and Definitions 

A. Reconsiderative Scoring; 

Reconsiderative test scoring occurs when a student's response to a multiple choice or completion 
(fiU-in-the-blank) question is reviewed by the teacher prior to the determination of correctness and point 
value. The reconsiderative method has been used for decades, even centuries, primarily in manual scoring 
of completion-style questions. Examples include cases where teachers award full or partial credit for 
misspelled responses, synonyms «,nd numeric calculations with minor errors. Also, incomplete responses, 
such as not distinguishing between John Adams and John Quincy Adams or writing only "homo" instead 
of "homo sapiens," can require reconsiderative scoring. The teacher maintains complete control by making 
professionally justifiable scoring decisions after the student responses have been collected and seen. 
Experienced teachers are thoroughly familiar with such practices. 

The reconsiderative method is partially subjective; the teacher is activel) making decisions based 
upo" his/her knowledge of the subject matter. However, the response type generally has the characteristics 
of objective assessments. When such scoring could be done only with manual methods, no special name 
was necessary. It was simply called fill-in-the-blank or completion testing. With machine-assistance now 
available, the use of the term reconsiderative test scoring appears to be appropriate. 

1 



A most simple explanation of machine-assistance for reconsiderative scoring starts by visualizing 
an item analysis displayed on a microcomputer monitor screen. (See Figure 1.) Each line gives the 
frequency tabulation and percentage ol[how ma ny students selecte d response A, B, C, etc. By moving 

r ^ — 



Respoase 


?oints 


Frequency 


Percentage 


A 




14 


11.7 


B 




35 


29.2 


C 




7 


5.8 


D 


2 


43 


35.8 


E 




21 


17.5 


TOTALS 




120 


100.0 



Figure 1: Frequency tat^ulation for reconsidering Hve responses of a traditional 
multiple choice question. 

the screen cursor (place designator) up and down the lines in an additional column called "Points", the 
teacher can enter from the keyboard the number of points each response should receive. For example, 
the B's could receive 3 points, the D's receive 1 point, and all others receive zero points. Then the 
computer re-scores every student's response and allocates the points that were designated by the teacher. 

The concept is quite simple, but its usefulness in the context of multiple choice questions 
appears, at first glance, to be minimal. However, reconsiderative scoring is founded upon fiU-in-the-blank 
(completion-style) questions, not on multiple choice. An item analysis tabulation for an un-cued (no 
options presented) completion question could look like Figure 2. (The free-response style question is also 
provided in the caption of the figure.) Such an item analysis has two major differences compared with the 
previous multiple choice example. 



Code 


Answer 


Pts. 


Sub. 


Freq. 


Percent 


000 


(Blank) 


0 


0 


2 


2.4 


186 


Canada 


2 


7 


1 


1.2 


307 


England 


1 


7 


6 


7.1 


325 


France 


0 


0 


1 


1.2 


328 


Great Britain 


0 


0 


4 


4.7 


412 


Ireland 


0 


0 


3 


3.5 


537 


United Kingdom 


2 


7 


67 


78.8 


562 


Yugoslavia 


0 


0 


1 


1.2 




TOTALS 






85 


100.0 



Figure 2: Example of a microcomputer display for reconsiderative scoring. A 
class of eighty-five students could be asked a free-response question: 
"Elizabeth II is the queen of what country?", with the following 
on-scrcen item analysis. The point values are designated by the 
teacher when moving the cursor up and down in the "Points" column. 
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1. The un-cued responses are not limited to a selection of five choices, and 

2. The actual word responses appear on the computer screen. 

When dealing with information as shown in Figure 2, the teacher can use his/her content 
knowledge in a reconsiderative way to evaluate each response. After reading the responses, the teacher 
designates the points earned by each. Then the teacher presses a key and the micro-computer does all the 
re-scoring. 



B. Computer-Assisted Scoring of Fill-In-The-Blank Responses 

But one might ask "How can we generate with existing technology an item analysis showing the 
actual word responses to completion (fill-in-the-blank) questions?" Such capabilities have been available 
since 1983 and are fully described in a 1987 book by Anderson. (This book is on ERIC microfiche; see 
item Al in the Annotated Bibliography.) The software program is called the MDT Educational Testing 
System. 

The MDT multi-digit method is essentially a fill-in-the-blank (completion-style) assessment that 
can be scored by machine. Appendix A provides examples, explanations and comments about prior 
research. Of special importance in Appendix A is the discussion of the improved feedback to teachers and 
students, including the actual words used to respond to each question. Variations of this relatively new 
cesting method have been called by different names: keylist testing, long-menu questions, un-cued 
responses, answer bank and multi-digit testing. 

(NOTE: Because of the importance of this method for the development of 
reconsiderative scoring, readers are encouraged to read Appendix A before 
proceeding.] 

Three recent independent research studies from Quebec (Brailovsky, Bordage, Allen and 
Dumont, 1988), Pennsylvania (Veloski, Rabinowitz and Robeson, 1988) and Ulinois (Anderson, 1988) 
support the use of the keylist/multi-digit/answer bank method. The research results indicate the following: 

1. Strengthened face validity (Veloski et al., 1988), 

2. Appropriate use in assessment of student's diagnostic skills (Brailovsky et al., 1988), 

3. Appropriate use with certain higher level problem-solving skills that cannot be tested by 
multiple choice questions (Veloski et al., 1988), 

4. Improved identification of marginal examinees (Veloski et al., 1988), 

5. General (but sometimes reluctant) acceptance by students (Anderson, 1988), 

6. Evaluative power perceived by students to be equal to that of fill-in-the-blank questions 
(Anderson, 1988), ^ 

7. Operational advantages over manually-scored handwritten short answer questions TBrailovskv 
et al., 1988), and ^ ^ 

I,' ^' 1^"*' reliability [of scanning] and economy when compared with multiple choice questions 
(VciOskj et 1' 88), 

The MDT innovation of machine-scored multi-digit testing is easily understood and can be used 
by htera ty hundreds of thousands of educators at all grade levels and in all academic areas. Early use 
ot the MDT innovation has o(«urred mainly in universities and medical schools. Subject areas range from 
art appreciation, military science and geography to mathematics and histology. Two national medical 
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accreditation examitiations (Family Practice and Opiithalmology) already are conducting pilot assessments 
with multi-digit long-list responses. Higli schools and junior highs can definitely benefit. For example, 
sophomore geometry final examinations in one high school have used the MDT format for three years. 
Elementaiy education may use it in modified forms, such as with two-digit responses. 

The lists (answer banks) and question banks could be prepared by individual users or by 
specialists in each field. These instructional materials could be disseminated to all and modified by anyone. 
Most of the terms for the lists come from indexes of textbooks, and the "MDT List Maker" software 
produces the appropriate data file. Participation by academic professional societies, textbook publishers, 
or sponsors of modest grants will greatly assist the preparation of these learning materials. 

Onto these capabilities of MDT multi-digit testing, the new features of reconsiderative scoring 
have been added. 



II. Initial Experiences with Computer-Assisted Reconsiderative Scoring 

Computer-assisted reconsiderative scoring was first available in an experimental software package 
called "RECON" in the Fall Semester of 1989. The author (Anderson) of this conference paper was the 
developer and first person to use RECON. One copy was sent to William Craig, Director of Testing of 
the Byron (Illinois) Cbmmunity Unit School District #226. Changes were made and the functional beta 
version of the software was distributed in mid-March 1990 to five additional experienced users of the 
keylist multi-digit method and software. Therefore, only Anderson's experiences in university classes and 
Craig's secondary school efforts are available to report at this time. 

A Synonyms and Multiple Correct Responses 

As described above, reconsiderative scoring allows for more than one correct response. Prior 
to havmg the RECON software, the recommended procedure with the multi-digit method was primarily 
to avoid such questions. However, occas'onally a manual review of the printed item analysis tabulation 
(showing word responses for each question) might reveal that two or more terms on the answer bank long- 
l'noJ°"*^ ^ considered correct. Then the teacher could look at a printed array (as shc;vn in Anderson, 
1987, p.99) to see for each student the full listing of the MDT code numbers in columns for each test 
question. By finding the MDT numbers of the additional correct answers, the points could be manually 
added to the student scores. 

Furthermore, because of this early limitation, the preparation of the answer bank lists either 
required extra caution to avoid synonyms or the use of duplicate code numbers for synonyms. Because 
of these difficulties, the use of questions with multiple correct responses was undesirable before the 
development of RECON. 

Although the removal of those difficulties was a major improvement, an even greater gain is the 
mcreased flexibility to generate truly chaUenging questions. Now, with reconsiderative scoring, svnonvms 
are desjrql^l^ in the answer banks. Diverse responses that communicate the same message of students' 
understanding can aU receive the same aUocation of points. Questions no longer need to be worded with 
so much precision to guide the students toward f\ preferred response. For example, one MDT lone-list 
question in a pre-RECON World Geography test asked the student "Give the name of the religion that 
has a pilgrimage known as the Hajj, but do not give the name of a believer of that religion." "Islam" is 
the correct answer, and the response "Moslem," a follower of the Lslamic religion, was not the desired 
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answer. To overcome this problem, reconsiderative scoruig permits the teacher to dijcide aij^r ihc 
responses are collected which answers vvUl receive IfuJl (or partial) credit. 

Crai ' (letter, 26 March 1990) illustrates the use of synonyms with a vocabulaiy item from the 
Byron high school sections of English I and American Liter atwre: "ITie _(blanfc)_ of a caterpillar into 
a butterfly is a wondrous process." Both 'transformation* and 'metamorphosis' are correct responses. 

B. Partial Credit 



The Islam/Moslem example above could easily result in partial cretdit being awarded if the 
teacher so decides. In other words, "close" can finally count for something. The teache* Jooks at the 
responses and then gives partial credit in the same manner that m civ^lit was givem. Tltis capability is 
exceUent to stimulate class discussion to distinguish between fully and jportially correct answers™ As an 
illustration, when Figure 2 was shown recently to a professor of cducatiow, the discussion quickly focused 
on the partial credit for the response "England." Two geography profes&tsrs debated with him that 
although Elizabeth II is the queen of England, the England of today is only or s part oil the country called 
the United Kingdom, as is Wales, etc. Because knowledge and thinking are more thM\ black/white and 
nght/wrong, the ability to easily award partial credit with machine assistance should fill a very useful niche 
m student assessments at all grade levels. 

C. Numeric Responses 

Three-digit numeric responses (not code numbers from answer banks) have been used since the 
beginnmg of muhi-digit testing in 1983. For example, 100 is the correct answer to "How many senators 
are there m the U.S. Cbngress?" (See Anderson, 1987, pp. 11, 32-33, 64-66, 82.) Care was needed to 
avoid questions with multiple correct answers. Now, greater flexibUity in representation and range of 
numeric responses can be permitted for reconsiderative scoring (as in Figure 3). A arecise number may 
or may not be required by the teacher, depending on the grade level and objectives of the class. This 



Code 


Response 


Pts. 


Sub. 


Freq. 


Percent 


004 




0 


0 


2 


6.5 


031 




0 


0 


1 


31.0 


042 




1 


4 


4 


12.9 


043 




2 


4 


13 


41.9 


044 




2 


4 


7 


22.6 


045 




1 


4 


2 


6.5 


114 




0 


0 


1 


3.2 


430 




0 


0 


1 


3.2 




TOTALS 






31 


100.0 



Figure 3: Reconsiderative scoring of numeric responses. A science laboratory 
exercise about measurement could ask the following question: To 
the nearest whole gram, what is the v/eight of the yellow precipitate 
in experiment J?" 



flexibaiKy h ©xtremcJy uselwl for questions such as How many milHfiM of people live in the USA in the 
iate 1980s? (For CMEnple, if your answer is «3 million, you should encode 083.)" For most classes, an 
»ns'H/er clo«j io 245 in correct. Plus or mrnus 3 million cowld get full credit. An answer of 235 or 255, 
although mcorr»cl, is certainly worth mortD than 172 or 426. The teacher decides. 



ILIk^lCOjj-CQDldbMQlL toM uUiple Choiice QuestiQns 

A, Towaid the Demise of m ADITIQNAL "multiple choice- 
Anderson has never been totally against multiple choice; he always envisioned the MDT answer 
bank method existing side by side with multiple choice to fulfill different objectives. But recent 
experiences with the new rec^nsiderative softivare capabiiiities have revealed how some limitations of 
traditional muUiplc choice qiiiestions can be overcome by the capabilities of RECON reconsiderative 
scoring. The advent of reconsiderative capabilities described below could be a serious challenge to the 
itraditional usage of multiple choice questions. 

IKiie term "multiple choice" appears to be a misnomer in its traditional usage where the student 
IS instructed to make a siijgle choice of one response out of five (y\BCDE) alternatives. 'ITiat traditional 
method should really be called "single choice from limited pool" questions. Note this important difference: 
A true "multiple" would be to sekct any combinatlon of the letters, such as CE or ACDE, from the same 
'iinited pool. There aie exactly thirty-two (32) possible combinations (see Figure 4) ranging from none 



101 


A 


106 


AB 


116 


ABC 


126 ABCD 


102 


B 


107 


AC 


117 


ABD 


127 ABCE 


103 


C 


108 


AD 


118 


ABE 


128 ABDE 


104 


D 


109 


AE 


119 


ACD 


129 ACDE 


105 


E 


110 


BC 


120 


ACE 


130 BCDE 






111 


BD 


121 


ADE 








112 


BE 


122 


BCD 


131 ABCDE 






113 


CD 


123 


BCVi 


(all) 






114 


CE 


124 


BDE 


132 (none 






115 


DE 


125 


CDE 


of them) 



Figure 4: Thirty-two possible combinations of five letters, each with an MDT 
multi-digit number. Any question with up to five alternatives labeled 
A, B, C, D, and E could be used with this special MDT list for i 
"multi-letier" responses. For example: "Which of the following 
characteristics is/are commonly associated with [whatever topic or 
situation the teacher chooses to present]: A) ...[word, phrase, 
sentence or even paragraph]... B) ... C) ... D) ... E) ... 

to all five letters. [Note: 32 = 2 raised to the fifth exponential power.] With four letters, there are 16 
combmati ns; three letters yield 8 combinations. For lack of a better name, this could be called either 
the 'multi-letter" format or the "power" format of responses. When each of the combinations is assigned 
a three-digit number, each combination becomes eligible for designation by students on an MDT multi- 
digit answer form for machine scoring. These responses would be scored using the reconsiderative 
methods, as discussed below. 

6 
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With this "multi-letter" format, questions with five statements can become more challenging. 
The process of elimination is no longer such a major factor. "Multiple-guess" is no longer one-jut-of-five 
(20%); blind guessing has an almost negligible one-out-of-thirty-two (3.125%) probability of picking the 
single best answer. 

Some educators might argue that this multi-letter approach has reduced the question to merely 
five True/False statements to be individually aixepted or rejected in the response. But close examination 
reveals that the same criticism aUso could be made of the single letter traditional version of multiple choice 
questions. The advocates of multiple choice questions have long felt that the use of five statements 
together is generally superior to five separate statements. 



B. Encouragement for Use of TRUE "Multiple" Choice 

One of the biggest unanticipated findings for Anderson was that reconsiderative scoring can 
make a major contribution to true multiple choice testing. The ease of using the RECON software brings 
renewed vigor to the use of questions with a selection of four or five responses. 

With the student responses sho*vn on the computer screen, the teacher proceeds to allocate an 
appropriate number of points to each of the multi-letter responses. If response "CE" is the best answer 
and is worth three points (and "ABD" is totally wrong), then what is the point value of "DE", or "BCDE"? 
Values of 3, 2, 1 or no points could be assigned. The teacher decides, making a professionally qualified 
decision with regard to the nature of the five statements. 

With the multi-letter format of responses, students can reveal more of their thinking. For 
example, if one of the five offered statements is patently incorrect, any student who includes that letter 
is indicating serious deficiencies. Likewise, any definitely correct options should not be excluded from the 
multi-letter response. Also revealing is the inclusion or exclusion of the other offered statements, either 
individually or in association with others. 

To facilitate this point allocation process, tables for manual assistance are being prepared to 
specify probable point values to cover most situations. The teacher's subjectivity is blended with 
professional competence to make the decisions. For example, the teacher could designate that alternative 
statement "E" has high weight for inclusion while statement "B" has high weight for exclusion in the multi- 
letter responses that receive the highest points. 

Although these computer-aided reconsiderative procedures are much faster than manual scoring 
of multi-letter responses, they do require more time to score '.han do the traditional single choice 
questions. However, that situation may change. The multi-lettev responses are highly compatible with 
computer-assisted item banks and test maker software. There are at least three options for developing 
these automated capabilities. First, for each multi-letter ("power") question, the item bank can easily store 
the information about which answers, i.e., combinations of letters, are to receive full credit and which merit 
partial credit. Second, the computer could be programmed to identify which table of partial points it 
would use with each item in the question bank. Third, each of the five statements could be assigned a 
value (based on difficulty or importance) and a computational algorithm could calculate the appropriate 
point value for any combination of letters. 

Some test developers (who write items, not software) are already engaged with true multiple 
choice quef tions. In the 1980s there has been an increase in the use of multiple choice questions that ask 
the student to "mark all that are correct". The annual state assessments in Illinois public schools have 

7 
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incorporated this format. This multiple mark approach has required the use of OMR optical mark readers 
(scamiers) that allow more than one mark in an answer grid. That requirement is highly contrary to the 
capabilities of high quality scanners to distinguish between light marks and poor erasures. In other words, 
the multiple mark educational measurement method is basically at odds with the hardware capabilities. 
The reconsiderative method as described above can provide TRUE multiple choice capabilities as good as 
or actually better than existing solutions that use multiple marks, unique wordings of the five choices, 
multiple answer keys or more costly scanners. 

C. Summary Comments about Multiple Choice Questions 

In recent years strong statements have been made both in attack and defense of multiple choice 
testmg in education. Much of the conflict could be attributed to the fact that the multiple choice format 
has been the only pencU-and-paper, machine-scored method of educational measurement capable of 
economically scoring millions of student responses while generating useful statistics. Multiple choice has 
not been a dead end street; it has been worthy of defense. But multiple choice is restrictive and 
contrived. It is not "natural;" the natural decisions in daily life are not based upon five choices, of which 
only one can correct. 

The origins of multiple choice testing (summarized in Anderson, 1988) were in the early 
Twentieth Century. Since then the method has been enhanced with research, statistics, machine scoring 
and some interesting applications, the best of which is probably adaptive assessment with tdch student at 
a computer terminal. The unquestionable greatest stren«»th of multiple choice testing is its ease of scoring, 
whether by hand or by optical mark readers. But the method is still primarily based on five choices that 
are more difficult to devise than the question itself with its correct answer. 

Enter the capabilities of reconsiderative scoring and multi-digit/answer bank responses. The 
answer bank method is founded upon fiU-in-the-blank/completion testing that is far older than the multiple 
choice method. And it can be scored by a machine! Likewise, the reconsiderative method is very old in 
concept but very new in machine-assisted applications. For example, the "multi-letter" variation discussed 
above was not thought of until the beginning of 1990. And major additional enhancements can be made, 
as discussed in Section V. 

Simply stated, the traditional multiple choice format is no longer the only serious contender for 
machine scoring of student responses, whether in small classes or in nationwide assessments. This is not 
a statement against multiple choice testing, which undoubtedly has a continuing niche to fill. This is a 
statement in favor of better education through a greater variety of more academically rigorous assessment 
methods. 



IV. Student Opinions 

Studem attitudes about MDT muhi-digit testing have been reported in eariier studies and are 
cited in Appendix A (Anderson, 1988, is the most recent and best summary of those findings). Anderson's 
classes tnat have been exposed to reconsiderative scoring total approximately 250 students. The students 
have each taker two or three examinations and have not indicated any difficulty in understanding how to 
respond to the new formats of questions. Nor have they considered the method unfair or inappropriate. 
Essentially, the students have attitudes that are neutral to favorable. They say that the MDT method plus 
reconsiderative scoring makes tests more difficuh, but fair. Further research with a short questionnaire 
of student opinions is planned. 



8 



ERIC 



10 



V. Additional Capabilities 

[NOTE: A detailed discussion of these capabUitics is scheduled as a keynote address 
by Anderson at the Second International Computer-Managed Learning Conference 
on 16-18 May 1990 in Edmonton, Alberta, Canada.] 

The MDT and RECON innovations are only the tip of an iceberg of software to enhance 
sducational measurement. Some of the items listed below could be developed quite quickly as additions 
io the existing software; others will require time, funding and perspiration. All gan be accomplished; we 
do not know if and when all should be developed. 

A. Multiple Responses to One Question 

As shown in Figure 5, one question can require several responses that could be given in any 
order. That example also illustrates use with higher order questions. 



Code Response 

017 Actinomycosis 

102 Bacterial Meningitis 

103 Bacterial Meningo- 
encephalomyelitis 



Pts. Sub. Freq. Percent 



0 
3 



0 
9 



18 
56 



22.5 
70.0 




Rabies 
Tetanus 

Thromboembolic Mcningc 
encephalitis 

TOTALS 



62 

560 



77.5 
700.0 



Figure 5: Complex medical diagnosis question: Questions 2-8: Give seven 
differential diagnoses for the following case. Data: Hereford, 650 
lbs., feedlot steer, vaccinated (IBR/BVD/PI3). Symptoms: Sudden 
onset of blind'-ss, tremors, frothy salivation, opisthotonos, gets 
better, then gel;. .. Drse. (Class size is 80 students, so 560 responses 
(7 X 80) are scored and tabulated.) 



Questions requirmg multiple responses to a single question can already be used with the existing 
SOP ^re, but care must be taken to verify manually on one printed report that the same answer was not 
given more than once by each studem. For example, 'Rabies' could be used only once, not seven times 
by the same student. That data check can be incorporated into the software. Also, in future versions the 
program will tally the multiple responses into one item analysis, hence the total being 300 percent for 
three responses or 700 percent for seven responses. 
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This multiple response capability is extremely powerful because questions can be phrased in so 
Mmy challenging ways. For example, some correct answers could be eliminated while also guiding the 
students to understand the nature of the question, as in: "Name two South American countries (other 
than Brazil) that have substantial areas of tropical rainforest." A multiple response question from the 
Byron school district (Craig, letter, 26 March 1990) was "List threo of the five characteristics of mammals." 

B. Enhanced Numeric Responses 

Response grids for longer numeric answers plus a machine readable decimal point are extremely 
important for mathematics and science education. An example, is in Figure 6. The solution for marking 
and scannmg will be compatible with the MDT ani RECON software and will utilize existing models of 
sheet and card readers. 



r 

Response Points in Sub. Freq. Percent 
SI S2 S3 



(Blank) 






564 


0.400 


.013 




1 


3,447 


2.700 


.04 






69 


0.001 


.13 




1 


7,431 


5.820 


.23 






36 


0.000 


.3 






843 


0.660 


.40 






16 


0.000 


.85 






73 


0.001 


1.2 


1 




4,087 


3.201 


1.3 


1 


1 


104,151 


81.569 


1.4 
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Figure 6: Simple numeric response in a standardized test. The wording of the 
question could be a story problem in which the student is to add 0.8 
plus 0.5 to test addition uf decimals. Note that the ability to add and 
the ability to place the decimal point are two separate tasks, for 
which students receive points under columns S2 and SI, respectively. 
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C. Multiple Steps in Responses 



Truly challenging questions are frequently complex, involving a ser:»3S of steps (intermediate 
products) to reach a final answer. The multi-digit responses (as numbers or as words) could be the input 
to equation solvers and concept mapping software. A reconsiderative capability could be used to assess 
multi-term and multi-step responses that reveal student thinking. This capability is BIG and could utilize 
artificial intelligence to assist teachers! 

D. Graphical Responses 

Scanning and reconsiderative grading of student graphical responses will open new dimensions 
for assessment. This capability will require different scanners, but the technology is already available and 
will include at least computer-recognition of handwritten numerals. 

E. Improved Feedback 

Better feedback to teachers, students and parents can utilize the already available "vocabulary" 
for correct and actual responses in the answer bank lists used with the MDT software. When coupled 
with improved statistical analyses (because guessing is almost eliminated), the improved feedback can assist 
adaptive instruction for remedial, regular and advanced study. 

F. Links with Computer Managed Learning and Databases 

Existing question banks (test generators) and administrative databases could be modified to 
utilize the completion-style responses of keylist/answer bank assessments. Data exchanges via networks, 
workstations and large file servers on micro, mini and mainframe computers will place major assessment 
and instructional power at the fingertips of teachers, and all steps can be very "user friendly." 

And more and more possible enhancements are yet to come. Anderson believes that he has 
only scratched the surface. Significant new applications and capabilities have become evident to him every 
semester for the past seven years. When more researchers with diverse backgrounds become involved, the 
pace of development is expected to accelerate. 



VI. Operational and Financial Issues 

The RECON and MDT software packages currently use common MS-DOS microcomputers. 
The more powerful microcomputers with 386 or 486 processors are recommended for professors with large 
classes. Reconsiderative scoring requires a fantastic number of calculations and data checks, so power and 
speed are highly desirable. Finahy, here is computer software that clearly justifies the acquisition of truly 
powerful microcomputers for educators. 

Standard answer form readers for sheets or cards are compatible. Almost any optical mark 
reader (OMR » scanner for test scoring) can be used, ranging from inexpensive (!; jOO.OO) manual-feed 
card readers to high-priced, high-speed sheet readers. The answer forms cost the same «»s those purchased 
for multiple choice tests. The entire system is designed to be available to and used by individual teachers 
or by centralized school offices for me^urement and evaluation. 

Although developed for classroom assessments conducted by individual teachers, the 
recomiderative method also could be applied to standardized, norm referenced tests. The ease of 
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generating test items with the MDT answer bank method will reduce the need for test security. In an 
educational environment where tests do at least influence th-^ curriculum, the ability to freely disseminate 
question banks as well as answer banks should have a favorable impact. 

Interested potential users should direct their inquiries to the various suppliers of scanners, 
answer forms, software and computers to obtain the most current information on supply, price and 
compatibility. 



VII. Educational ImpHcatioP *- and Research Issues 

The advent of reconsiderative scoring (as concept and as functional software) should stimulate 
both research in the concept and the reporting of applied experiences with the software. Numerous 
topics of inquiry are evident, not the least of which are the relationships to test theory. 

A Relevance to Assessment Theory 

A fundamental question is this: "How do the MDT and RECON capabilities support or conflict 
with theories of education?" 

Robert J. Mislevy's work on "Foundations of a New Test Theory" (1990) includes the following 
insights: "Tomorrow's tests must present tasks that learners in the different states [of competencjr] are 
likely to carry out in observably different ways. We cannot limit out interest to the correct response, but 
must also consider factors such as speed, intermediate products, and incorrect responses. We must also 
examine the patterns of similarity or dissimilarity acrc&s tasks that probe knowledge structures or problem- 
solving techniques, llic new test theory must provide models that can express these patterns." 

At least at first glance, the characteristics of multi-digit answer bank questions plus the 
reconsiderative scoring capabilities appear to be supportive of Mislevy's desired models and methods. This 
support could be especially important when working with large numbers of students that make at least 
some machine assistance an economic necessity. 

Numerous theoretical concerns need to be addressed in future presentations. The paragraphs 
below highlight a few of the educational implications and topics appropriate for further research. 

B. Reduction of Bias 

Because each item analysis is a tabulation of the studer* responses, the computer makels certain 
Uiat each and every student with response "XXX" receives precise.^ me same number of designated points. 
This eliminates inconsistencies and bias that can occur in manual scoring of completion questions when 
the teacher spends hours to go from the top to the bottom of a stack of examinations. 

C. Ease of Writing Questions 

The educational benefits of the reconsiderative Liulti-digit format include the ease of writing 
questions that do not need four wrong but plausible foils. When used with diagrams such as those in 
Figure 7, hundreds of questions are easily generated. 
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D. Increase in Academic Rigor 

A third major benefit is the increased academic rigor over similar multiple choice questions. 
If a discrete answer to a question is known, calculated or derived by thought, then the desired response 
can be located easily in the alphabetized list. But if not known or derived, the correct response is only 
one out of many, and not one out of five. Guessing is senseless; the process-of-elimination is applicable 
only if the student has studied sufficiently to have the appropriate vocabulary and concepts in mind. 




Figure 7: Diagrams compatible with MDT-style questions. 



E. Application ibr Instruction as Well as Assessment ' 

The MDT format plus reconsiderative scoring can be used for tests, exercises and homework, 
whether for grades or as learning experiences. The promise held forth by the MDT software and the 
RECON module is not merely on the assessment side, it seems to me, but on the instructional side as 
well. Using this software to assess our students' mastery of local goals and objectives imparts an 
intensified classroom focus on those goals and objectives" (Craig, letter, 26 March 1990). 

F. Inclusion of Subjective Input 

The RECON method allows teachers to have subjective input where appropriate. 
Reconsiderative scoring could stimulate greater usage of higher order questions. 
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O. Ease of Understanding 

All indications arc that machine-assisted reconsiderative sccring can be easily understood and 
utilized by teachers and students in upper elementary, secondaiy and pi.ist-secondary education, including 
vocational and professional training. Being veiy natural, like manual re&msiderative scoring, the method 
could have wide acceptance, as discussed below. 



Vin. Co mments on Adoption and Dissemination 

A. The Underlying Conceptual Model 

Being so simple, the RECON capability almost defies being called an innovation. But it is a 
classic example of how innovative ways can ease the burden of age-old tasks. Teachers have been doing 
reconsiderative grading of responses one-by-one for centuries. They undoubtedly understand the task. 
Nov^ with computer speed and ease to let teachers virtually see all student answer sheets at the same time 
in tabulated foim, teachers do faster and better the same job that they already know and do so well. Info- 
WQdsi Magazine (13 February 1989, page 1) discusses three factors needed for a truly user-friendly 
software, interface: the screen, the command structure, and "even more important, ... the underlying 
conceptual model of the software." The spreadsheet concept made VisiCalc and Lotus 1-2-3 great 
programs for this reason: "experienced financial analysts can pick up the program in 15 minutes, because 
they understand the underlying task so weU." Likewise, teachers already understand the task of reconsi- 
derative scoring. RECON is software that teachers can readily use. 

B. The Problem of Inertia 



On the other hand, the biggest threat to the practical application of the MDT and RECON 
mnovations is the inertia in all levels of America's schools. Instead of lamenting that stumbling block to 
all innovations, we can explore three ways to overcome the inertia that hinders the adoption of computer- 
assisted reconsiderative scoring. 

1- fiwt way is through Anderson whose company (MDT Corp.) can influence the 
availability of the MDT/RECON software. The software already exists with commercial quality and 
includes standard capabilities for multiple choice and criterion referenced testing. Anderson's difficult 
double role as a businessman as well as an academic innovator is not the topic of this academic paper, but 
that role is an important facto/ for keeping the software affordable for the maximum number of schools. 

2. The second way focuses on getting microcomputers and essential software into the 
hands of teachers. Anderson believes that fully integrated reconsiderative scoring, as described in Section 
V above, can become for teachers what word processors are for secretaries and writera, spreadsheets are 
for accountants, and databases are for managers. And the reason is this: If a teacher and his/her students 
are to receive the discussed benefits, the teacher (or assistant) is actually required to look at the 
microcomputer screen and use arrow and number keys to interact with the student responses. The only 
alternative method for reconsiderative scoring is slow manual grading. And that manual scoring looses the 
benefits of computer-generated feedback to improve student learning and question quality. TVpewriters, 
pocket calculators, card indexes, and even stand-alone multiple-choice grading machines have either 
allowed many teachers to avoid using computers or have allowed some education administrators to say that 
Jtudents, not teachers, should get the microcomputers. Instead, all teachers should have microcomputers 
jnd essential software (including reconsiderative scoring) for their daily tasks. 
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3. The third way to overcome inertia in education is strongly influenced by organizations 
and corporations with a vested interest in the advancement of computer-assisted education. They can 
have major impact upon what actually becomes known to and accepted by educators. However, they can 
also compound the problem of inertia when their self-proclaimed leadership roles as "champions of 
education" are merely lip-service while their true roles are those of followers of profit, as discussed below. 



IX. Initial E?q>eriences with External Sponsors 

The inertia in the "champions of education" is clearly shown in the initial experiences at the 
corporate level concerning reconsiderative scoring. MDT Corporation has made disclosures of the 
RECON concept and capabilities to numerous entities m the past year. The objective of the disclosures 
was to find a sponsor, partner, or advocate to assist in the development of reconsiderative scoring methods. 
These entities included IBM, Zenith, National Computer Systems (NCS), ScanTron, HEI Scanning Systems, 
ETS, ACT, several major textbook publishers, US Department of Education, state education agencies in 
Illinois, and major software developers like MicroSoft and Lotus. To date, every one has adopted either 
a "We don't do that" attitude or a low-risk, "market-driven", "not-invented-here", wait-and-see attitude to 
determine where profits (not educational improvements) can be found. This is not a criticism of these 
entities; it is only a clear statement of the obstacles ♦hat confront any innovation in education. 

What John Roach (Chairman of Tandy Corporation) said in 1984 about the microcomputer 
jndustiy is quite applicable to the issues of educational technology: "We're in an industry where promotion 
is more important than the technology." Sad, but true: Without recognition and promotion, innovations 
cannot attain practicality. 

The bright side is that if educators can show that a sufficient market does exist, one or many 
of the corporate and not-for-profit entities would gladly and efficiently carry the banner of educational 
innovation and even reform. However, inertia and conservatism among educators and educational 
measurement specialists could lead to very slow or no acceptance of computer-assisted reconsiderative 
scoring. To paraphrase some comments from the above named entities, "Teachers don't want this stuff. 
They don't know how to use it, and many are afraid of computers. This looks like more work, not less. 
Besides, students don't want tougher tests, and parents don't like being shown how little their kids know. 
It won't sell easily, and we are not in the education reform business. But we'll be glad to work with you 
when the market is evident." 



X. Conclasion 

Although the initial in-coarse experiences with reconsiderative scoring are quite favorable and 
encouraging, no conclusions can be made on the merits of the new method until more users have results 
to report. The willingness of research-minded educators to examine objectively the issues associated with 
reconsiderative scoring is crucial. The author firmly believes that the method will withstand the most 
vigorous scrutiny. Furthermore, with additional users, still more innovative ways to enhance the initial 
capabilities will be discovered. This present (April 1990) presentation is intended to introduce and 
stimulate discussion about one additional measurement technique made possible by educational technology. 
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APPENDIX A: Bnckground to MDT Multi-Digit Testing 



In 1982 a geography professor, Dr. Paul S. Anderson, returned to the USA after fourteen years 
in Australia and Latin America. For the first time he was faced with large classes needing computer- 
assisted testing. The unwelcome necessity to use multiple choice tests prompted him to seek a more 
rigorous alternative. In four months he conceptualized and made operational a method of computer- 
assisted scoring of fil!-in-the-blank (completion-style) questions. Anderson's geography training in 
cartograpliy, remote sensing, computer analysis, and regional studies gave him, respectively, the necessaiy 
baclcground to print answer sheets, understand electronic scanners, appreciate computer power and utilize 
alphabetized long lists of terms, as in an atlas gazetteer. What emerged has far exceeded his initial 
expectations and has led him on a lateral career path to explore and develop the potential of his initial 
innovation. After seven yeans of extra hours, heavy investment of personal funds, and even times of 
anguish, Anderson has produced two fully operational innovations for educational measurement, and 
further innovations can be clearly seen. 

The innovation for machine scoring of fill-in-the-blank questions that Anderson first used in 
early 1983 is described in his book, The MDT Innovation (1987, available in all libraries with ERIC 
microfiche). The 200-page book discusses the method's origins, initial usage, applications and education- 
al implications, including financial savings and relevance to higher order learning. 

II. Method 



The MDT innovation is so straight-forward that it can be explained in a single diagram (Figure 
8) found on the book's cover. 

A. Completion-style questions are asked in a wide variety of ways, including some that require 
numeric responses. 

B. With an answer in mind, the student locates his/her desired response on an alphabetized 
iong-hst "answer bank" appropriate for the subject area. The list is quite important, but it can be made 
?asily by any teacher or shared by teachers of similar courses. 

C. The three-digit code number of the response is marked on an answer form that is read by 
standard optical mark readers of sheets or cards. 

D. Microcomputer software processes the student responses, issues their scores and prints 
distinctly useful reports. i 



m. Disting uishing Differences from Prior Efforts 

With such a simple and natural methodology, was nothing like this ever tried previously? 
Chapter 8 in Hnt. MPT InnQv^tjpp (Anderson, 1987) reviews prior efforts fortunately not known to 
Anderson when he began. Anderson's work is distinctive in two crucial ways. 
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Figure 8: Examples of MDT-style questions, lists and answer forms. 



4 

First, no previous researcher had significantly gone beyond one hundred responses, that is, 
responses with two digits. Andereon began his work with three^igit numbers that allow up to 1(X)0 terms 
and concepts in the answer bank. Each list is intentionally long to nearly eliminate the Til recognize thr 
answer if I see it" effect. Short lists ei jourage and permit searching that waste time when students try 
to recognize answers. This is a fundamental difference from "matching" tests. List length should be in 
consideration of the grade level of the students, but totally at the discretion of the test writer. For 
university students, Anderson recommends lists of over 500 terms. Furthermore, he uses response number 
999 to mean "no reasonable answer is on this list." When responding with 999, the student also writes the 
correct word response in a blank space on the answer sheet. The evident strength of the MDT method 
is the high degree to which it emulates completfon-stylc (fiU-in-the-blank) responses, of which the academic 
rigor and validity are well established. 
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Second, earlier researchers did not develop any computer program to do the scoring. 
Consequently, Anderson was the first to observe and use the tremendous time-saving advantages of these 
X)mputer-scored completion tests. Even more important, he recognized and made operational the 
:apability to generate some truly distinctive feedback for teachers and students. The most innovative 
leedback involves the ability to print the actual word responses of the students and of the teacher's answer 
key. The MDT responses with printed words in the Individual Student Report (Figure 9) give far more 
assistance to the student than do the multiple choice responses of A B C D or E The Item Analysis 
tabulation of responses (Figure 10) gives valuable information to the teacher about student learning. 
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Figure 9: Individual student report Figure 10: Item analysis with tabulation 

with MDT actual word responses. of student word responses. 

Even though he worked with these reports regularly since 1983, Anderson needed nearly four 
years before the second innovation became evident to him. Only in the 1989-90 academic year has this 
second innovation become a reality in a software program called RECON for reconsiderative scoring. 
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NOTES: 

1. This bibliography is divided in several ways. The publications and presentations of Paul S 
Anderson, the originator of MDT multi-digit testing and RECON, are presented by types (books, articles, 
etc.) m chronological order. The works of others are in a separate section in alphabetical order. 

2. The designation "ERIC identifies items available internationally in many libraries via indexes 
and microfiche. Copies are available from the Educational Resources Information Center, c/o ERIC 
Document Reproduction Service, Alexandria, VA 22304-6409. 

3. All readers are requested to nominate additional items for inclusion in this bibliography. 
Please send references (and copies, if available) to: » r 



Dr. Paul S. Anderson 
Department of Geography-Geology 
Illinois State University 
Normal, IL 61761 

Phone (309) 438-7360 (Office) 
(309) 438-7649 (Department) 
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Comments in Al. 
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Cbmments in Al. 

G7. Veloski, J. Jon, Howard K. Rabinowitz, M.D. and Maiy R. Robeson (1988). "Cueing in 
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of the Association of American Medical Colleges, Chicago, 11-17 November, 1988, pp. 195-200. 

Highly relevent research. "The results support the feasibility of large group administration of tests 
conducted in an open-ended format that can be scored by computer. Not only is this format equally 
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Verbal Aptitude Tests," Applied Psychological Measurement Vol. 6, No. 1, Winter 1982, pp. 1-11. 

Uses term "keylist." Short lists of up to 100 choices. No computer program for scoring. Additional 
comments in Ai. 
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With some reproducible materials.) (Group C.) 

Cl. (1983) "Introduction to the Large-List 'Multi-Digit Test* Procedure: With Examples From 
a Worid Geography Course," West Lakes Regional Conference of the Association of American 
Geographers, Iowa City, Iowa, October 1983. 

See Al and Bl. 

C2. (1984a) "Applications of the Multi-Digit-Test (MDT) Procedure for Teaching the Geography 
of Latin America," Conference of Latin Americanist Geographers (CLAG), Ottawa, Canada, September 

See B2. i 

C3. (1984b) "The Multi-Digit Test Procedure: Refinements and Preliminary Results," 
International meeting of the National Council for Geographic Education, Toronto, Canada, October 1984 
with co-authors M. Hill, S. Naim and W. Walters. 

See Al and B3. 
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C4. (1985a) "Laboratory Schools as a Unique iJetting for Research: The Experimentation with 
the Multi-Digit Test (MDT) at Illinois State University High School," with co-author Eileen Kanzler. 
National Association of Laboratory Schools convention in C jnver, Colorado, 26-28 February 1985. 

Mainly about the laboratory school setting. 

C5. (1985b) "Comparison of Cognitive Achievement in Objective Testing: Multi-Digit and 
Multiple Choice Tests," with co-author Eileen Kanzler. Conference of the American Educational Research 
Association (AERA), Chicago, Illinois, 4 April 1985. (ERIC ED 260 131) 

Major paper. Much was incorporated into Al. This paper led to the nationwide press coverage in Dl. 

C6. (1985c) "Applications of the Multi-Digit Test (MDT) in Geography," Demontstration 
presented to the Conference of the National Council for Geographic Education, Breckenridce. Colorado. 
5-9 August 1985. 
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List Tests, Subjective Question Scores and Instant Scoring in the Classroom," Discussion session at the 
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Illinois Science Teachers Association convention in Normal, Illinois, 4-5 October 1985. 

C9. (1986a) "Multi-Digit (MDT) Testing in the Teaching of Criminal Justice Sciences," with co- 
author Diane Alexander, Academy of Criminal Justice Sciences, Oriando, Rorida, March 1986. f ERIC ED 
282 936) ^ 

CIO. (1986b) "Applications of the MDT Multi-Digit Testing Method for Cartographic Information 
Education," Annual meeting of the North American Cartographic Information Society, Philadelphia, 
Pennsylvania, 15-18 March 1986. ^ 

Cll. (1987a) "Answer Banks and Question Banks for Immediate Classroom Use: MDT Materials 
1?11 Aprfu987"'* ^'^^ Studies," Annual meeting of the Illinois Geographical Society, Elgin, lUinois, 

C12. (1987b) "Student Attitudes about MDT Multi-Digit Testing Analyses from Pioneer 
Experiences," Annual meeting of the National Council on Measurement in Education, held jointly with the 
American Educational Research Association, Washington, DC, 19-23 April 1987. (ERIC ED 296 000) 

Early results later used in B5 and C15. 

C13. (1987n and 1988) "Changing World Patterns of Machine-Scored Objective Testing: The 
Expected Impact of the Multi-Digit Method," with co-author Alcyone Saliba, Sixth World Congress of 
Comparative Education, Rio de Janeiro, Brazil, 6-10 July 1987. Also presented to annual conference of 
the Midwest Comparative and International Education Society, Normal, Illinois, 19-20 February 1988. 

See B7. 



24 

26 



C14. (1987b) "Advantages of MDT Multi-Digit Testing in Veterinary Medicine Education: Basic 
Features," (with separated demonstration). Fifth Symposium on Computer Applications in Vete-inary 
Medicine, Urbana, liUnois, 26-29 September 1987. 

C15. (1987c) "Comparison of Stude. Attitudes about Seven Formats of Educational Testing, With 
Emphasis on the MDT Multi-Digit Testing lechnique," Annual meeting of the Mid-Western Educational 
Research Association, Chicago, Illinois, 15-17 October 1987. (ERIC ED 295 999) 

Major paper incorporated with C12 into B5. 

C16. (1989) "Introduction to Machine-Assisted Reconsiderative Test Scoring: A new method for 
partial credit and multiple correct responses," Special presentations to medical and science educators, 
Philadelphia, PA, 11-12 October 1989. 



MEDIA ITEMS : (Group D.) 

Dl. Associate Press (national). May 1985, various titles including "New Exam Takes Guessing 
Out of Multiple-Choice Test." 

Created awareness nationwide, including commentaries by radio announcers. 

D2. Pqpt^graph (McLean County, IL), August 31, 1986, page A3, "ISU Professor Devises Testing 
Method to Make Guessing a Thing of the Past." 

Profile of Dr. Anderson as professor, innovator and businessman. 

D3. ^mn^ tQ Pusin^ (McLean County, IL) Februaiy 1987, pp. 24-27, "Computerization Takes 
Drudgery Away From Teachers." 

Profile of Dr. Anderson's activities. 

Technological Horizons in Education fT.H.E.1 Journal (national), March 1987, pp. 46-47, 
"Teacher's Testing Method Eliminates Guessing on Machine Scored Exams." 

This is perhaps the best single-page discussion of MDT methods and usage prior to 1987. 
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Brief Introduction to Computer- Assisted Reconsiderative Scoring: 
A New Method for Partial Credit and Multiple Correct Responses 

Paul S. Anderson, Ph.D., Dept. of Geography-Qeology, lilinois State Univerlty, Normal, IL 61761 

MPT Corporation, 107 Broadway, Normal, IL 61761 (309) 4S2-6388 

Recoosklerative test Koring occun when a litudent's response to a multiple choice or completion (fill-in-the-blank) question is read 
by the teacher prior to the determinatioa of correctness and point value. Historically, reconsiderative methods have been used extensive^ 
in manual scoring of completion>style questions. Examples include full or partial credit for synonyms, misspelled responses, numeric 
calculations with minor errors, 44id incomplete answers. The teacher has complete control. The method incoi.rroraies the best aspects of 
objective and subjecth« assessmenu because the teacher uses his/her content knowledge and professional Judgement when making decisions 
after reading the student's response. 

Computer'assisted reconsiderative scoring uses a special item ana^is tabulation of responses (Figure 1). The actual word responses 
are shown on the computer screen. By moving the computer cursor (place designator) down the lines in the column called "Points", the 
teacher can enter from the keyboard the number of points each response should receive. Then the microcomputer program called RECON 
(tm) does all the renscoring to allocate poinu to each student. 

The ability to generate tabulations showing the actual word responses to completton questions has been available since 1983. (See 
The MDT Innovatton. 1987, by Anderson on BRIC microfiche ED 307 287.) An alphabetized "answer bank" with up to 1000 terms is a 
distinguishing component. Each tcm has an MDT multi-digit identifier that can be marked on answer forms for machine reading. The 
"answer bank" lists can be prepared by individual teachers or by progressive textbook publishers and professional societies. 

Three independent research studies in 1988 (abstractt available) support the use of this relatively new MDT testing method. The 
research resulu indicate 1) strengthened face validity, 2) appropriate use with problem solving and diagnoses, 3) improved identification of 
marginal examinees, 4) acceptance by students, and S) evaluathne power equal to that of fill-in-the-blank questions. Other operational 
advantages include ease of writing questions, reduction of bias, and improved feedback to students and teachers. The MDT method offers 
greater academic rigor because it eliminates recognition associated with multiple choice cued responses. Hie MDT multi-digit method is 
essentially machine-scored fill-in-the-blank assessment, especially when used with the RECON capabilities that became available in 1990. 

In addition to the major advantages of reconskierathre scoring and awarding partial credit, RECON also permiu the following 
innovative features: questkms requiring multiple responses (Figure 2); calculated numeric responses (Figure 3); "multi-letter" responses in 
whteh every combination (such as ACD) of the five cued foils may be a response for full, partial or no credit (Figure 4). Future enhance- 
ments can accommodate decimal points, multiple steps, graphical responses and much more. 

All indications are that computer-assisted reconsiderathw scoring can be easily understood and utilized by teachers of all subject 
matter in schools ranging ftom upper elementary through college, medical schools and vocational training. By allowing teachers to have 
subjective input where appropriate, reconsiderative scoring shoukl stimulate greater usage of higher order questions. When progressive 
teachers utUize these new capabilities and when "education corixirations" (for scanners, textbooks and tests) provide the essential sponsorship 
for research and usage, the MDT and RECON innovations could make a major contribution to the improvement of American education. 



QKk 


Aniwer 


Pis. 


Sub. 


Frcq. 


Perceni 


000 


(BUnk) 


0 


0 


2 


14 


1B6 


Canada 


2 


1 


1 


U 


307 


EoflaMl 


1 


1 


6 


7.1 


32S 


Fiance 


0 


0 


1 


i2 


m 


Orcai Britain 


0 


0 


4 


4.7 


412 


IrcUnd 


0 


0 


3 


3.3 


337 


Uniied Kincdom 


2 


7 


67 


78J 


362 


Yugqilavui 


0 


0 


1 


1.2 




TOTALS 






85 


loao 



Cbde 


Response 




Sub. 


Frc4|. 


Pi^fceni 


017 


Aciinomycntk 


0 


0 


18 


213 


102 


BacieiUI Menln|i'iii 


3 


9 


56 


7ao 


m 


Bacleiial Meningo- 












cncephaloinyelilU 


0 


0 


21 


26.3 


806 


Rabiea 


3 


9 


77 


96.3 




Teunua 


0 


0 


5 


6.3 


916 


Thromboembolic MeninfO* 












encephalilii 


3 


9 


62 


77.5 




TOTALS 








7oao 



Figure 1: Eaampte of a mkrocompuicr dispby for lecomlderatWe scoring. A 
t\m of cighiy-Ove siudcnu could be aiked a frec^icspocuc queslion: 
"Eliubclh n it ibc queen of whal counlry7*i wiib Ibe following 
on*screen iiem analysis. Tbe poini values are deiignaied by ihe 
leacbcr wben moving ihe cunor up and down in the "PoinU* column. 



Figure 2: Complex medical diagnosis question: Questions 2-8: Oive seven 
differeniial diagnoses for the following case. Data: Hcteford. 630 
lbs., feedkM steer, vaccinated (IBRyBVD/P13). Symptoms: Sudden 
onset of blindness, tremors* frothy salivation, opiithoionos, gets 
better, then geu worse. (Class sixe is 60 studcntSi so 360 responses 
(7 X HO) are scored and tabulated.) 
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Rgure): Rccomlderaliveseoring of numeric responses. A seleoce laboratory 
esTf dse about measMfjemem could ask the following questioo: *To 
the nearest whole grami whal ii the weight of the yellow piccipitaie 
kt experiment Jt* 



Figure 4: TUrty«lwo possible combinatkNis of five fetteri, each whh ao MDT 
multi-digit number. Any question with up to five alternatives labeled 
A, B. C D. end B eouM be uied with thk special MDT Hit for 
*multi*leiier* responscL fof example: *Whicb of the folk»»te| 
characterisiies is/ere eoesmonty assodaied with (whatever topic or 
situatkw the teacher chooses lo present]: A) .^|word, pbrest, 
sentence or m«ft nariiiaohL. h\u. Cl ^ ^ Bl ^ 



